League of Legends Win Prediction with XGBoost

This notebook uses the Kaggle dataset League of Legends Ranked Matches which contains 180,000 ranked games of League of Legends starting from 2014. Using this data we build an XGBoost model to predict if a player's team will win based off statistics of how that player played the match.

The methods used here are applicable to any dataset, we use this dataset to illustrate how SHAP values help make gradient boosted trees such as XGBoost interpretable because it is large, has many interaction effects, contains both categorical and continous values, and the features are interpretable (particularly for players of the game). For more information on SHAP values see: https://github.com/slundberg/shap

In [1]:
import pandas as pd
import numpy as np
import xgboost as xgb
from sklearn.model_selection import train_test_split
import shap
import matplotlib.pyplot as pl

shap.initjs()

Load the dataset

To run this yourself you will need to download the dataset from Kaggle and ensure the prefix variable below is correct.

In [2]:
# read in the data
prefix = "data/league-of-legends-ranked-matches/"
matches = pd.read_csv(prefix+"matches.csv")
participants = pd.read_csv(prefix+"participants.csv")
stats1 = pd.read_csv(prefix+"stats1.csv", low_memory=False)
stats2 = pd.read_csv(prefix+"stats2.csv", low_memory=False)
stats = pd.concat([stats1,stats2])

# merge into a single DataFrame
a = pd.merge(participants, matches, left_on="matchid", right_on="id")
allstats = pd.merge(a, stats, left_on="matchid", right_on="id")

# Convert string-based categories to numeric values
cat_cols = ["role", "position", "version", "platformid"]
for c in cat_cols:
    allstats[c] = allstats[c].astype('category')
    allstats[c] = allstats[c].cat.codes
    
X = allstats.drop(["win"], axis=1)
y = allstats["win"]

# create train/validation split
Xt, Xv, yt, yv = train_test_split(X,y, test_size=0.2, random_state=10)
dt = xgb.DMatrix(Xt.as_matrix(),label=yt.as_matrix())
dv = xgb.DMatrix(Xv.as_matrix(),label=yv.as_matrix())

Train the XGBoost model

In [3]:
params = {
    "eta": 0.2,
    "max_depth": 4,
    "objective": "binary:logistic",
    "silent": 1,
    "base_score": np.mean(yt),
    "eval_metric": "logloss"
}
model = xgb.train(params, dt, 300, [(dt, "train"),(dv, "valid")], verbose_eval=25)
[0]	train-logloss:0.618377	valid-logloss:0.618588
[25]	train-logloss:0.32616	valid-logloss:0.327287
[50]	train-logloss:0.27976	valid-logloss:0.281052
[75]	train-logloss:0.257905	valid-logloss:0.25939
[100]	train-logloss:0.244147	valid-logloss:0.245882
[125]	train-logloss:0.234861	valid-logloss:0.236798
[150]	train-logloss:0.227573	valid-logloss:0.229895
[175]	train-logloss:0.221406	valid-logloss:0.223929
[200]	train-logloss:0.216545	valid-logloss:0.219345
[225]	train-logloss:0.211733	valid-logloss:0.214854
[250]	train-logloss:0.207822	valid-logloss:0.211104
[275]	train-logloss:0.20418	valid-logloss:0.207639
[299]	train-logloss:0.201126	valid-logloss:0.20487

Explain the XGBoost model

Because the Tree SHAP algorithm is implemented in XGBoost we can compute exact SHAP values quickly over thousands of samples. The SHAP values for a single prediction sum

In [4]:
# compute the SHAP values for every prediction in the validation dataset
shap_values = model.predict(dv, pred_contribs=True)

Explain a single player's chances of winning a particular match

SHAP values sum to the difference between the expected output of the model and the current output for the current player. Note that for the Tree SHAP implmementation the margin output of the model is explained, not the trasformed output (such as a probability for logistic regression). This means that the units of the SHAP values for this model are log odds ratios. Large positive values mean a player is likely to win, while large negative values mean they are likely to lose.

In [5]:
shap.visualize(shap_values[0,:], feature_names=Xv.columns, data=Xv.iloc[0,:])
Out[5]:
Visualization omitted, Javascript library not loaded!
Have you run `initjs()` in this notebook? If this notebook was from another user you must also trust this notebook (File -> Trust notebook). If you are viewing this notebook on github the Javascript has been stripped for security.
In [83]:
xs = np.linspace(-4,4,100)
pl.xlabel("Log odds of winning")
pl.ylabel("Probability of winning")
pl.title("How changes in log odds convert to probability of winning")
pl.plot(xs, 1/(1+np.exp(-xs)))
pl.show()

Summarize the impact of all features over the entire dataset

A SHAP value for a feature of a specific prediction represents how much the log odds prediction changes when we observe that feature. In the summary plot below we plot all the SHAP values for a singel feature (such as goldearned) on a row, where the x-axis is the SHAP value (which for this model is in units of log odds of winning). By doing this for all features, we see which features drive the model's prediction a lot (such as goldearned), and which only effect the prediction a little (such as kills).

In [6]:
shap.summary_plot(shap_values, Xv.columns, max_display=20)

Examine how changes in a feature change the model's prediction

The XGBoost model we trained above is very complicated, but by plotting the SHAP value for a feature against the actual value of feature for all players we can see how changes in the feature's value effect the model's output. Note that these plots are very similar to standard partial dependence plots, but they provide the added advantage of displaying how much context matters for a feature (or in other words how much interaction terms matter). How much interaction terms effect the importance of a feature is capture by the vertical dispersion of the data points. For example earning only 5,000 gold during a game may lower your logg odds of winning by 6 for some players or only 3 for others. Why is this? Because other features of these players effect how much earning gold matters for winning the game. Note that the vertical spread narrows once you earn 20,000 gold, meaning the context of other features matters less for high gold earners than low gold earners.

The y-axis in the plots below represents the SHAP value for that feature, so -4 means observing that feature lowers your log odds of winning by 4, while a value of +2 means observing that feature raises your log odds of winning by 2.

In [84]:
# sort the features indexes by their importance in the model
# (sum of SHAP value magnitudes over the validation dataset)
top_inds = np.argsort(-np.sum(np.abs(shap_values), 0))

# make SHAP plots of the three most important features
for i in range(3):
    shap.plot(Xv.iloc[:,top_inds[i]], shap_values[:,top_inds[i]], Xv.columns[top_inds[i]], alpha=0.01)

Using color to highlight interaction effects

Here we use and interaction_plot version of the SHAP plots to color the datapoints with another feature that most explains the interaction effect variance. For example earning less gold is less bad if you have not died very much, but it is really bad if you also die a lot.

Note that these plot just explain how the XGBoost model works, not nessecarily how reality works. Since the XGBoost model is trained from observational data, it is not nessecarily a causal model, and so just because changing a factor makes the model's prediction of winning go up, does not always mean it will raise your actual chances.

In [116]:
# make interaction SHAP plots of the ten most important features
for i in range(40):
    
    # zoom the plot in past the any extreme outliers
    pl.xlim(np.percentile(Xv.iloc[:,top_inds[i]], 0.01), np.percentile(Xv.iloc[:,top_inds[i]], 99.99))
    pl.ylim(np.percentile(shap_values[:,top_inds[i]], 0.01), np.percentile(shap_values[:,top_inds[i]], 99.99))
    
    shap.interaction_plot(top_inds[i], Xv, shap_values, alpha=0.01) # using alpha help show density

Train a model on just totminionskilled to see marginal association

Some of the plots above are unexpected, such as the lower chance of winning the more minions you kill. It is important to note that the above plots show the effect of a feature (such as totminionskilled) in the context of on all the other features. So given that you have earned a lot of gold, killing more minions means you must have gotten that gold elsewhere. To just look at the impact of knowing how many minions a player killed we can train an XGBoost model with just totminionskilled as the only feature.

In [39]:
ind = np.where(Xv.columns == "totminionskilled")[0][0]
Xt_min = Xt.iloc[:,ind:ind+1]
Xv_min = Xv.iloc[:,ind:ind+1]
dt_min = xgb.DMatrix(Xt_min,label=yt.as_matrix())
dv_min = xgb.DMatrix(Xv_min,label=yv.as_matrix())
params = {
    "eta": 0.1,
    "max_depth": 4,
    "objective": "binary:logistic",
    "silent": 1,
    "base_score": np.mean(yt),
    "eval_metric": "logloss"
}
model_min = xgb.train(params, dt_min, 300, [(dt_min, "train"),(dv_min, "valid")], verbose_eval=25)
[0]	train-logloss:0.692026	valid-logloss:0.691947
[25]	train-logloss:0.689362	valid-logloss:0.689074
[50]	train-logloss:0.689135	valid-logloss:0.688902
[75]	train-logloss:0.688968	valid-logloss:0.688776
[100]	train-logloss:0.688868	valid-logloss:0.688693
[125]	train-logloss:0.6888	valid-logloss:0.688647
[150]	train-logloss:0.688742	valid-logloss:0.688618
[175]	train-logloss:0.688695	valid-logloss:0.688583
[200]	train-logloss:0.688654	valid-logloss:0.688561
[225]	train-logloss:0.68862	valid-logloss:0.688536
[250]	train-logloss:0.688592	valid-logloss:0.688519
[275]	train-logloss:0.688575	valid-logloss:0.688507
[299]	train-logloss:0.68856	valid-logloss:0.688498
In [45]:
shap_values_min = model_min.predict(dv_min, pred_contribs=True)
In [76]:
pl.xlim(0,400) # focus the plot where most of the data is
pl.ylim(-1,1) 
shap.plot(Xv_min.iloc[:,0], shap_values_min[:,0], Xv_min.columns[0], alpha=0.01)

Train a model on just wardsplaced to see marginal association

In [101]:
ind = np.where(Xv.columns == "wardsplaced")[0][0]
Xt_wards = Xt.iloc[:,ind:ind+1]
Xv_wards = Xv.iloc[:,ind:ind+1]
dt_wards = xgb.DMatrix(Xt_wards,label=yt.as_matrix())
dv_wards = xgb.DMatrix(Xv_wards,label=yv.as_matrix())
params = {
    "eta": 0.1,
    "max_depth": 4,
    "objective": "binary:logistic",
    "silent": 1,
    "base_score": np.mean(yt),
    "eval_metric": "logloss"
}
model_wards = xgb.train(params, dt_wards, 300, [(dt_wards, "train"),(dv_wards, "valid")], verbose_eval=25)
[0]	train-logloss:0.692808	valid-logloss:0.692823
[25]	train-logloss:0.69132	valid-logloss:0.691459
[50]	train-logloss:0.691236	valid-logloss:0.691391
[75]	train-logloss:0.691208	valid-logloss:0.691373
[100]	train-logloss:0.691192	valid-logloss:0.691365
[125]	train-logloss:0.691185	valid-logloss:0.691364
[150]	train-logloss:0.691182	valid-logloss:0.691362
[175]	train-logloss:0.691181	valid-logloss:0.691361
[200]	train-logloss:0.691181	valid-logloss:0.691361
[225]	train-logloss:0.69118	valid-logloss:0.691362
[250]	train-logloss:0.691179	valid-logloss:0.691361
[275]	train-logloss:0.691178	valid-logloss:0.69136
[299]	train-logloss:0.691178	valid-logloss:0.691361
In [102]:
shap_values_wards = model_wards.predict(dv_wards, pred_contribs=True)
In [103]:
pl.xlim(0,60) # focus the plot where most of the data is
pl.ylim(-0.5,0.5)
shap.plot(Xv_wards.iloc[:,0], shap_values_wards[:,0], Xv_wards.columns[0], alpha=0.01)

There are of course many more plots one could build to explore this dataset and how XGBoost models it. The notebook is available at: https://github.com/slundberg/shap